Determination of the effectiveness of two methods for usability evaluation using a CPOE medication ordering system

Objectives: To assess the effectiveness of two usability evaluation methods, cognitive walkthrough (CW) and think aloud (TA), for identifying usability problems and to compare the performance of CW and TA in identifying different types of usability problems. Methods: A CWwas performed by two usability evaluators and 10 physicians were recruited to perform a TA usability testing of a CPOE system (Medicator). The severity of identified usability problems was determined and the usability problems were categorized based on the User Action Framework (UAF). The potential of usability problems to cause medication errors was also determined. The thoroughness, validity and effectiveness of the two methods were compared. Results: Fifty seven unique usability problems of different severity, spread over the four phases of interaction as defined by the UAF, were identified. The effectiveness of the TA method for identifying usability problemswas 0.08 higher than that of theCW(0.70 vs. 0.62). The thoroughness (the extent to which a method can identify existing usability problems) of the TA was higher for the “Planning” and “Assessment” phases and lower for the “Translation” phase (as defined by UAF). The thoroughness of TA for identifying problems that may potentially result in medication errors was higher than that of CW (0.81 vs. 0.68). The number of usability problems identified by each of the methods was significantly less than the total number of detected real usability problems in Medicator (p < 0.001). The observed differences between the number of real usability problems identified by CW and TA (38 vs. 41), the difference between the average severity of the detected problems by CW and TA (2.37 vs. 2.41), and the difference for identifying problems potentially resulting in medication errors (15 vs. 18) were not statistically significant (p > 0.4). Conclusions: This study shows that although TA showed a slightly better effectiveness, there is no significant difference between the performance of the CW and the TA methods in terms of number of usability problems identified and the mean severity of these problems. Since no single evaluation method will uncover all of the usability problems a combination of methods is advised as the most appropriate approach, especially if usability problems can lead to potentially fatal outcomes

Clinicians satisfaction with CPOE ease of use and effect on clinicians’ workflow, efficiency and medication safety

Objectives: To study the satisfaction of end-users of a computerized physician order entry (CPOE) system concerning ease of use and the effect on users’workflow, efficiency, and medication safety and to seek users’ opinions regarding required improvements of the system. Usability evaluation had shown that this system, which was in use for almost a decade, contained a number of severe usability problems. So another objective of the study was to determine whether there was a direct relation between user satisfaction and the results of a usability evaluation of the system. Methods: Two survey questionnaires were distributed to CPOE system users (physicians and nurses) working in inpatient departments of a university hospital. Questionnaires included items thatwere rated using a five point Likert scale. Multiple choice questions with space for free text additions also were used to collect qualitative data concerning the use of the CPOE system and the users’ opinion concerning system requirements for improvement. Datawere analyzed using descriptive statistics and by the use of Mann–Whitney U and KruskalWallis tests. Results: Two hundred seventeen physicians and 587 nurseswere eligible to participate in this study (response rate 49% and 56% respectively). Physicianswere satisfied with the CPOE ease of use (median 3.8, interquartile range [IQR] 3.3–4), and the effect on workflow (median 3.7, IQR 3.3–4), medication safety (median 3.75, IQR 3–4), and efficiency (median 4, IQR 3–4). Nurses also had a positive attitude towards CPOE ease of use (median 3.6, IQR 3–4), and its effect on workflow (median 3, IQR 3–3.6), medication safety (median 3, IQR 2.5–3.5), and efficiency (median 3.5, IQR 3–4). Users mainly indicated that the system needs: supplementary functionalities (e.g. alerts for allergies), improvement of current functionalities, integration with other hospital information systems and improvement of information presentation (e.g. a clear medication overview). Users did not use some current functionalities because of lack of awareness of the functionalities or having difficulty in using them. Conclusions: Users of this CPOE system, which was used for almost a decade, were satisfied with the system’s ease of use and its effect on efficiency, workflow and medication safety although the system showed many usability problems and lacked some functionalities. In this case study, therefore, there seems no direct relation between the results of the earlier performed usability evaluation and user satisfaction as determined in the current study

Effect of predefined order sets and usability problems on efficiency of computerized medication ordering

Objectives: To study the effect of predefined order sets on the efficiency of computerized medication ordering, and to analyze the effect of different types of usability problems on ordering efficiency. Methods: Crossover study to comparing the efficiency of two methods of ordering (with and without use of predefined order sets) in a laboratory setting using a computerized physician order entry system (CPOE). The excess number of mouse clicks and keystrokes (the difference in number of mouse clicks and keystrokes needed by each physician and the minimally required numbers to accomplish the ordering tasks) for each method was measured and per physician, occurrences of usability problems during the task sessions were recorded. Observed usability problems were categorized using Zhang et al.’s heuristic principles of good user interface design. The effect of different types of usability problems on the excess number of mouse clicks and keystrokes was statistically analyzed. Results: The median excess number of mouse clicks and keystrokes needed by physicians was 6.2 times lower in the method with predefined order sets (p < 0.01). The excess number of mouse clicks and keystrokes was significantly increased by vague and erroneous system messages with a factor of 2.62 (95% CI 2.24–3.07), the use of unfamiliar language and terminology by a factor of 1.28 (95% CI 1.14–1.43), and non-informative system feedback by a factor of 1.15 (95% CI 1.03–1.28), respectively. Other categories of usability problems had little influence on ordering efficiency. Conclusions: Predefined order sets can improve the efficiency of computerized ordering by reducing the excess number of mouse clicks and keystrokes. However, the efficiency of computerized ordering can be significantly impaired by usability problems due to vague and incorrect system messages, unfamiliar language, and non-informative system feedback

Context Sensitive Health Informatics: Concepts, Methods and Tools

__Abstract__ Context is a key consideration when designing and evaluating health information technology (HIT) and cannot be overstated. Unintended consequences are common post HIT implementation and even well designed technology may not achieve desired outcomes because of contextual issues. While context should be considered in the design and evaluation of health information systems (HISs) there is a shortcoming of empirical research on contextual aspects of HIT. This conference integrates the sociotechnical and Human-Centered-Design (HCD) approaches and showcases current research on context sensitive health informatics. The papers and presentations outlines theories and models for studying contextual issues and insights on how we can better design HIT to accommodate different healthcare contexts

A usability study to improve a clinical decision support system for the prescription of antibiotic drugs

Objective A clinical decision support system (CDSS) for empirical antibiotic treatment has the potential to increase appropriate antibiotic use. Before using such a system on a broad scale, it needs to be tailored to the users prefer

Has the Rate of CD4 Cell Count Decline before Initiation of Antiretroviral Therapy Changed over the Course of the Dutch HIV Epidemic among MSM?

Introduction:Studies suggest that the HIV-1 epidemic in the Netherlands may have become more virulent, leading to faster disease progression if untreated. Analysis of CD4 cell count decline before antiretroviral therapy (ART) initiation, a surrogate marker for disease progression, may be hampered by informative censoring as ART initiation is more likely with a steeper CD4 cell count decline.Methods:Development of CD4 cell count from 9 to 48 months after seroconversion was analyzed using a mixed-effects model and 2 models that jointly modeled CD4 cell counts and time to censoring event (start ART

Non-AIDS defining cancers in the D:A:D Study-time trends and predictors of survival : a cohort study

BACKGROUND:Non-AIDS defining cancers (NADC) are an important cause of morbidity and mortality in HIV-positive individuals. Using data from a large international cohort of HIV-positive individuals, we described the incidence of NADC from 2004-2010, and described subsequent mortality and predictors of these.METHODS:Individuals were followed from 1st January 2004/enrolment in study, until the earliest of a new NADC, 1st February 2010, death or six months after the patient's last visit. Incidence rates were estimated for each year of follow-up, overall and stratified by gender, age and mode of HIV acquisition. Cumulative risk of mortality following NADC diagnosis was summarised using Kaplan-Meier methods, with follow-up for these analyses from the date of NADC diagnosis until the patient's death, 1st February 2010 or 6 months after the patient's last visit. Factors associated with mortality following NADC diagnosis were identified using multivariable Cox proportional hazards regression.RESULTS:Over 176,775 person-years (PY), 880 (2.1%) patients developed a new NADC (incidence: 4.98/1000PY [95% confidence interval 4.65, 5.31]). Over a third of these patients (327, 37.2%) had died by 1st February 2010. Time trends for lung cancer, anal cancer and Hodgkin's lymphoma were broadly consistent. Kaplan-Meier cumulative mortality estimates at 1, 3 and 5 years after NADC diagnosis were 28.2% [95% CI 25.1-31.2], 42.0% [38.2-45.8] and 47.3% [42.4-52.2], respectively. Significant predictors of poorer survival after diagnosis of NADC were lung cancer (compared to other cancer types), male gender, non-white ethnicity, and smoking status. Later year of diagnosis and higher CD4 count at NADC diagnosis were associated with improved survival. The incidence of NADC remained stable over the period 2004-2010 in this large observational cohort.CONCLUSIONS:The prognosis after diagnosis of NADC, in particular lung cancer and disseminated cancer, is poor but has improved somewhat over time. Modifiable risk factors, such as smoking and low CD4 counts, were associated with mortality following a diagnosis of NADC

Prototypes of computer-assisted instruction for arithmetic word-problem solving: a training study on improving the ability of educable mentally retarded children to solve simple addition and subtracti

Contains fulltext : mmubn000001_118130307.pdf (publisher's version ) (Open Access)Promotores : J. Dumont en E. van Lieshout205 p

Identifying and eliminating inefficiencies in information system usage : A lean perspective

Objectives: Mismatches frequently occur between information system (IS) dictated workflows and actual workflows of IS users. The resulting impeded workflows negatively influence the efficiency with which goods or services are produced and delivered to customers. Within a healthcare context, impeded workflows can additionally have a negative impact on the safety and effectiveness of care delivered to patients. Methods: This study evaluates the impact of an electronic health record system of a large university hospital on workflows of healthcare professionals from a lean management waste perspective. Workflow mismatches were identified from direct observations and follow-up semi-structured interviews with physicians and nurses and assessed in terms of waste generated. Results: In total, 241 manifestations of waste were identified and classified according to 8 types of waste. Furthermore, noteworthy relationships among the identified manifestations of waste were found. These include differences in frequency per type of waste, certain types of waste having a cascading effect, and waste proliferating as a result of mimicked and routinized behavior. Conclusions: The knowledge obtained from this study can support (re) design of IS to better match workflows which may subsequently lead to more safe, effective and efficient patient car